Dynamical Motifs: Building Blocks of Complex Network Dynamics 
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Spatio-temporal network dynamics is an emergent property of many complex systems which re- 
mains poorly understood. We suggest a new approach to its study based on the analysis of dynamical 
motifs - small subnetworks with periodic and chaotic dynamics. We simulate randomly connected 
neural networks and, with increasing density of connections, observe the transition from quiescence 
to periodic and chaotic dynamics. We explain this transition by the appearance of dynamical motifs 
in the structure of these networks. We also observe domination of periodic dynamics in simulations 
of spatially distributed networks with local connectivity and explain it by absence of chaotic and 
presence of periodic motifs in their structure. 
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Dynamics in networks underlie functioning of many 
complex systems such as the brain Q , cellular regulatory 
machinery 0, ecosystems [j| and many others. These 
systems exhibit a wide repertoire of dynamics, ranging 
from periodic oscillations in cell cycle and brain rhythms 
to chaos in food webs and chemical reactions. Despite the 
recent rapid advancements in our ability to elucidate sta- 
tistical properties of the underlying networks 0, IU, El ' 
surprisingly little is understood about their dynamical 
behavior. This is due to several reasons, in particular, 
inadequacy of the methods of nonequilibrium statisti- 
cal mechanics in the domain of heterogeneous mesoscopic 
systems and inability of the dynamical systems theory to 
deal with systems having more than order- 1 dimensions. 

Fundamental problem which one faces while try- 
ing to understand dynamics in complex networks is 
the strong influence of their structure on their non- 
Hamiltonian dynamics. This influence may induce 
long term connectivity-dependent spatio-temporal corre- 
lations which present formidable problem for understand- 
ing of the dynamics. Statistical methods allow to solve 
this problem in the limit of infinite-size networks , but 
they are not applicable to the study of realistic networks 
with non-uniform connectivity and a relatively small size. 

It was recently found that many real networks include 
statistically significant subnetworks, so-called motifs, in 
their structure In this Letter we suggest the use of 
dynamical motifs - small subnetworks with non-trivial 
dynamics - as a new approach to the study of recur- 
rent dynamics in complex networks. In it we combine 
dynamical and statistical methods to identify dynami- 
cal motifs and evaluate probability of their occurrence 
in the structure of networks. We show that the emer- 
gence of periodic and chaotic dynamics in networks of 
increasing structural complexity is linked to the appear- 
ance of periodic and chaotic motifs in their connectivity. 
We also consider spatially distributed networks with lo- 
cal connectivity and show that chaotic motifs are absent 
in their structure. We also suggest that this approach 
may be useful for study of the dynamics in networks of 



arbitrary structure and size. 

In many complex systems the dynamics of individual 
elements and the rules of their interaction are relatively 
simple and the resulting complex behavior is an emer- 
gent consequence of these interactions. Hence, in order 
to study the influence of the structure on the dynamics 
of networks let us focus on models with simplest inter- 
actions and dynamics at each node. Let Xi(t) £ [0;1], 
i = 1, . . . , N be a set of variables describing properly 
scaled states of N elements connected in a network. 
Consider the time evolution of network's state vector 
X(t) = {xi(t),X2(t), . . . ,xjv(t)} described by the follow- 
ing set of first order differential equations 



dX(t) 
dt 



-X(t) + F(X(t)), 



(1) 



where F(X) = {fx(X), f 2 (X), . . . , f N (X)} is a set of 
sigmoid nonlinearities with [0; 1] value ranges. This gen- 
eral class of models includes continuous version of Ran- 
dom Boolean (Genetic) Networks (cRBN) [j. Ho|. in 
which fi(X) are randomly chosen Boolean functions of 
their arguments, and continuous-time Artificial Neural 
Networks (cANN) in which fi(X) = f((W ■ X) l + 
<7j), where W is the coupling matrix and <jj are thresh- 
olds. Both of these models were shown to exhibit complex 
periodic and chaotic dynamics in the biologically relevant 
cases of intermediate probabilities of gene expression in 
cRBN 10] and non-symmetric interactions in cANN Q. 

To illustrate the use of dynamical motifs we employ 
a simple cANN model with f(x) = (1 + exp(— 20 x))' 1 , 
uniform external excitation a, t = a = 0.5 and inhibitory 
interactions of the same strength: W = —Gw, where 
w = 5 and G is the adjacency matrix of the directed 
graph on which the network is defined. In this setting the 
model is similar to the simplified version of a balanced 
network model with excitatory connections replaced 
by a uniform field and can be viewed as a simple model of 
a cortical microcircuit. It is also an extension of the con- 
cept of winnerless competitive networks ^3 to the case 
of arbitrary connectivity. However, methods presented 
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FIG. 1: Fraction of networks F with either periodic or chaotic 
(A) and only chaotic (■) dynamics in simulations of 200-node 
cANN networks with node-to-node connection probability p. 
Lines represent predictions based on the probability of occur- 
rence of periodic (dashed line, Eq. |0) and chaotic (solid line, 
Eq. ©) motifs. 



in this Letter can be used for other models as well. 

We have performed Monte Carlo simulations of the 
described above cANN model defined on an ensemble 
of random networks with N — 200 nodes and uniform 
probability p of node-to-node connections, i.e., an Erdos- 
Renyi (ER) ensemble. A sample of 2 • 10 4 p random net- 
works was generated for each considered p and cANN dy- 
namics was simulated 100 times on each of the networks, 
each time starting with different initial condition taken 
at random from the hypercube R^pL. Sets of initial con- 
ditions were considered in order to eliminate the influ- 
ence of the basins of attraction in multistable networks 
which in itself is a complicated issue requiring separate 
research. Largest Lyapunov exponent A was calculated in 
each simulation. Networks with at least one initial con- 
dition leading to A 6 (-0.005; 0.005), typically A - 10~ 4 
were classified as having limit cycle dynamics and with 
A > 0.005, typically A ~ 10 _1 as having chaotic dynam- 
ics. As the probability of connections p was increased, 
the transition from fixed point to periodic and chaotic 
dynamics was observed around p — 0.015 (Fig. ^ trian- 
gles) . The transition to solely chaotic dynamics occurred 
around p = 0.025 (Fig-QJ squares). 

We now apply the concept of dynamical motifs in or- 
der to explain such observations and make further pre- 
dictions about the dynamics in networks. The main idea 
behind this approach is that the transition to periodic 
or chaotic overall dynamics in a network occurs due to 
the appearance in its structure of small, not necessary 
isolated, subnetworks which have the same type of dy- 
namics. We call these subnetworks dynamical motifs. Of 
course many dynamical motifs may be present in a given 
dynamical network, but at least one is needed in order 
for the network to have a given type of dynamics. The 
dynamical phase transitions that are observed in models 




FIG. 2: Smallest dynamical motifs with periodic (a) and 
chaotic (b-d) dynamics. Triangles indicate direction of the 
inhibitory connections. 



of complex networks are then identified with the perco- 
lations of dynamical motifs, i.e., 0-to-l transitions in the 
probability of their occurrence. 

Let us consider an ER network with the probability of 
node-to-node connection p. The probability p n that some 
n nodes of this network form a given motif with I links 
and no self-loops is equal to p l (l — p)"(™ _1 ) _ '. It is pos- 
sible that some of the nodes in the motif are suppressed 
by connections from a number / of nodes outside of a 
motif that are frozen in the " + 1" state. Such connec- 
tions would suppress dynamics of the motif and should 
be ruled out. By excluding them we obtain corrected 
probability 



Pnf =P 1 (1 ~P) 



n(n+/-l)-i 



(2) 



It is difficult to calculate exactly the probability to find 
N(n;i) such motifs in a network. However, approximate 
calculation of the probability P( n -i) to encounter one or 
more motifs is straight forward and is expected to work 
well in the case of sparsely connected networks (p << 1): 



P(n-l) = 1 - P(N {n , l} = 0) « 1 - (1 - p nf ) **=5J1 
PnfNl 



1 — exp 



A(N -n)\ 



(3) 



where Nl/(A(N — n)\) is the number of ways to pick n 
nodes of the motif from the iV-node network multiplied 
by n\/A - number of ways to label a motif, with A being 
the order of motif's automorphism group. This formula 
predicts percolation of these motifs at some intermediate 
value of p which depends on n and I. We are interested in 
the values of n and I for which the percolation occurs at 



the smallest p = p c . Let us define it by the point where 



(n;Q 



1/2. Then 
AH 

(AT-n)!A 



Pi(l-Pc) 



log 2. 



(4) 



Assuming N 3> n and p c <C 1 we find that p c ~ 
y^i/'TV - ™/'. To minimize p c , n// should be maximized 
and hence subnetworks with most nodes and least links 
will appear first as p is increased from 0. 

Subnetworks with limit cycle dynamics include 3-loop 
(Fig. Ufa)), 4-loop and other, more complicated, struc- 
tures. Since for the n-loop I — n and A = n, 3-loop has 
smallest p c . According to Q), its probability of occur- 
rence is given by 



-P(3;3) ~ 1 - exp 



3 y V 



p) 



3(1+/) 



(5) 



and is plotted in Fig^by a dashed line (with / evaluated 
from simulations). This estimate predicts the appearance 
of dynamics in ER networks very accurately. 

In order to find chaotic motifs we used the nauty pack- 
age to generate all possible non-isomorphic directed 
graphs with up to 8 nodes and 11 links, simulated the 
cANN dynamics on them and calculated the largest Lya- 
punov exponent A for each. Digraphs with A > 0.005, 
typically A ~ lO^ 1 were classified as chaotic. In Fig.^b- 
d) we show the first three chaotic motifs in the order of 
increasing number of nodes (5, 6 and 7 nodes) and mini- 
mal number of links (9, 10 and 10 links). Also, we found 
six non-isomorphic chaotic motifs with 8 nodes and 11 
links, four of them with A — 2 and two with A = 1. Nu- 
merical evaluations of p c according to J3J indicate that 
the latter motifs have smallest p c as 0.025. Probability to 
find one or more such motifs in an ER network is given 
by 



Pn 



1 -exp(-47V 8 p 11 (l -P) 



45+8/\ 



(0) 



and is piloted in Fig. 2] by a solid line (with / evalu- 
ated from simulations of chaotic networks). As can be 
seen from the plot, this prediction is reasonably good. 
The discrepancy may be caused by the approximate na- 
ture of the estimate (0 , severe undersampling of the net- 
work space in simulations and disregard of the fact that 
not only frozen, but also periodically oscillating external 
nodes suppress chaos in these motifs. 

While the origin of periodic dynamics in n-loops is ob- 
vious as they merely are negative feedback loops, the 
nature of chaos in chaotic motifs is not self-evident. We 
have traced the route to chaos in the first chaotic motif 
by lowering w to 0.5 and then gradually increasing it. 
In Fig. |31 the bifurcation diagram of the local maxima 
of xi(t) oscillations is plotted. This diagram reveals two 
period doubling cascades, one starting around w = 0.7 
and another one around w = 2.5. 




FIG. 3: Bifurcation diagram depicting local maxima of xi(t) 
oscillations for a range of strengths w of the inhibitory con- 
nections in the first chaotic motif (Fig.lTfh)). Period doubling 
cascades start around w = 0.7 and w = 2.5. 



However, there are no conclusive experimental obser- 
vations of chaotic dynamics in either genetic or neuronal 
networks. Presence of chaotic dynamics would be incon- 
sistent with the requirements of robustness and repro- 
ducibility of the response imposed on the living organisms 
by the environment. On the other hand, periodic dynam- 
ics in these networks are very common. Hence, some of 
the assumptions that were used in these simplified mod- 
els must be wrong. As we show below, one of them is 
the assumption of uniformly random connectivity. Re- 
cent experimental data suggests that metabolic networks 
possess scale-free structure while neuronal networks 
are highly clustered on both small and large 
spatial scales. Moreover, neurons in the brain frequently 
form ordered spatial structures with distance-dependent 
probabilities of connections, so-called cortical microcir- 

cuits mm 

To illustrate the influence of spatial structure on the 
dynamical properties of networks we simulated the cANN 
model on the 12-by-12 two-dimensional square lattice 
with neuron-to-neuron connection probabilities obeying 
Gaussian distribution and forbidden self-connections: 



p{dij) = KN(1 



6ij)e~ 



■) / £ (1 - 6 mn )e~(^) 2 

m,n=l 

where dij is a metric distance between neurons i and j, 
7 is the length scale of the distribution, K is the av- 
erage number of connections per neuron and 5{j is the 
Kronecker delta. In effect, 7 controls clustering of the 
connectivity, with values close to 1 corresponding to net- 
works with mostly local connectivity and large values 
effectively diminishing the role of spatial structure and 
corresponding to ER-like connectivity. 

We generated a random sample of 1000 such networks 
with 7 = 2 and K = 14. As in the case of an ER sam- 
ple, cANN dynamics was simulated in each of the net- 
works for 100 different random initial conditions and the 
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largest Lyapunov exponent was calculated in each simu- 
lation. An average connectivity K = 14 corresponds to 
p k 0.1 which has led to approximately 99% of chaotic 
networks in a sample of ER ensemble (Fig.^b)). On the 
contrary, around 99% of the networks with 7 = 2 exhib- 
ited periodic dynamics and only about 1% were chaotic. 
This result indicates that clustering plays an important 
role in defining dynamical properties of neural networks. 
It may support an observation that many real neuronal 
networks are locally clustered and exhibit reproducible, 
except for stochastic effects, dynamics. 

We now show how the idea of dynamical motifs may be 
used to understand these observations. Let us enumerate 
grid nodes by 1, . . . , N and consider a motif with n nodes 
and I links placed on the grid. We label its nodes by 
^i,^2; ■ ■ ■ ; and its links by i ai , i a2 , . . . , i ai ib, with 
djbj, j = i . . .1 denoting ordered pairs of nodes that are 
connected. Probability Pi a i b for the grid nodes i a and ib 
to be connected is distance-dependent and is given by 10. 
Then the average probability that n nodes of a network 
form this motif is obtained by averaging over all possible 
placements of its nodes on a grid: 



Pn 



1 N II s 

w £ II'' II ' 



by choosing connectivity that maximizes the probability 
of motifs with desirable dynamics and minimizes proba- 
bility of motifs with unacceptable dynamics. Using this 
method we predict that connectivity of cortical microcir- 
cuits might be such that it minimizes the occurrence of 
chaotic motifs in their structure. 

It was shown in |8( that many real networks have 3- 
loops in their structure. In most cases there are very few 
of such loops and it was argued that their presence is not 
statistically significant. However, we suggest that such 
dynamical motifs are important because presence of even 
one or two of them may profoundly influence dynamical 
behavior of the whole network by slaving dynamics of 
many adjacent nodes. 
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(8) 



fe=i 



where s = n{n — 1) — I and UkVk are all the 
pairs of unconnected nodes. For example, an av- 
erage probability for a 3-loop (Fig. EJa)) is p3 = 
irV^ P- • P- ■ P- ■ (1 — P- ■ )C\ — P 



A 



,)(1 



-Pijja). As expected, in the limit of distance-independent 
probabilities expression © becomes equivalent to the ER 
case. Probability of occurrence of one or more motifs in a 
network can again be approximated by J3J . It is approx- 
imately 1 for 3- and 4- loops and P m ss for the (5; 9) 
motif in networks with A = 2. Other chaotic motifs would 
be even less probable because connection probability falls 
off quickly with distance in these networks. Also, P m w 1 
for the (5; 9) motif in networks with A = 12. Hence, 
periodic motifs are present and chaotic motifs are ab- 
sent in spatial networks with local connectivity (A = 2), 
but chaotic motifs are present in non-local networks with 
A = 12. These calculations may also explain recent ob- 
servations of periodic and chaotic dynamics in models of 
cortical neural microcircuits with local and non-local spa- 
tial organization of connectivity 0] . Eq. (JSJ) may also be 
applied in the study of networks with other distributions 
of connectivity. 

In conclusion, a method to study dynamical behav- 
ior of networks by examining minimal building blocks of 
the dynamics was suggested. Calculations of abundance 
of dynamical motifs in networks with different structure 
allow to study and control dynamics in these networks 
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